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Reading Mastery 

Program Doscription^ Reading Mastery, one of several curriculum components that 

constitute the Direct instruction curriculum from SRA/McGraw- 
Hill, is designed to provide systematic instruction in reading to 
students in grades K-6. Reading Mastery, which can be used as 
an intervention program for struggiing readers, as a supplement 
to a school’s core reading program, or as a stand-alone reading 
program, is available in three versions: 

1. Reading Mastery Ciassic (for use in grades pre-K-3) aims to 
help beginning readers identify letter sounds, segment words 
into sounds, blend sounds into words, develop vocabulary, 
and begin to learn comprehension strategies. 

2. Reading Mastery Plus (for grades K-6) has a language-arts 
focus with an emphasis on reading, writing, spelling, 

and language. 



Ressarch^ One study of Reading Mastery that falls within the scope of meets WWC evidence standards with reservations. The studies 

the Adolescent Literacy review protocol meets What Works included 361 students in grades 4 and 5, who attended schools 

Clearinghouse (WWC) evidence standards, and one study in the midwestern and northwestern United States.® 

1. The descriptive information for this program was obtained from a publicly available source: the program’s website (http://www.mcgraw-hill.co.uk/sra/ 
downloads/Reading%20Mastery/Reading%20Mastery%20Signature%20Edition%20Brochure.pdf, downloaded October 2009). The WWC requests 
developers to review the program description sections for accuracy from their perspective. Further verification of the accuracy of the descriptive infor- 
mation for this program is beyond the scope of this review. The literature search reflects documents publicly available by August 2009. 

2. The studies in this report were reviewed using WWC Evidence Standards, Version 2.0 (see the WWC Procedures and Standards Handbook, Chapter III), 
as described in protocol Version 2.0. 

3. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 



3. Reading Mastery Signature Edition (for use in grades K-5) 
includes three strands: (a) the Reading strand addresses pho- 
nemic awareness, phonics, word analysis, fluency, vocabulary, 
comprehension, spelling, decoding, and word recognition 
skills; (b) the Oral Language/Language Arts strand addresses 
oral language, communication, and writing skills; and (c) 
the Literature strand is designed to provide students with 
opportunities to read a variety of different types of text and to 
develop their vocabulary. 

During the implementation of Reading Mastery, students are 
grouped with other students at a similar reading level, based on 
program placement tests. The program includes a continuous 
monitoring component. 
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Research^ (continued) Based on two studies, the WWC considers the extent of reservations examined the effectiveness of Reading Mastery 

evidence for Reading Mastery on adolescent learners to be on adolescent learners in the alphabetics or general literacy 

small for the reading fluency and comprehension domains. No achievement domains, 
studies that meet WWC evidence standards with or without 



EffactivaneSS Reading Mastery was found to have potentially positive effects on reading fluency and no discernible effects on comprehension for 
adolescent learners. 



Rating of effectiveness 
Improvement index'^ 



General literacy 

Alphabetics Reading fluency Comprehension achievement 

na Potentially positive No discernible effects na 

effects 



na +19 percentile points Average: -7 na 

percentile points 

na na Range: -20 to +7 na 

percentile points 



na = not applicable 



Additianal program 
information 



Developer and contact 

Reading Mastery was originally called DISTAR [Direct instruction 
System for Teaching Arithmetic and Reading). Early versions of 
Reading Mastery were developed during the 1960s and 1970s 
by Dr. Siegfried Engelmann as part of the Direct Instruction 
teaching model.® Reading Mastery is distributed by SRA/ 
McGraw-Hill, 220 East Danieldale Road, DeSoto, TX 75115- 
2490. Web: www.sraonline.com. Email: SRA_CustomerSer- 
vice@mcgraw-hill.com. Telephone: (201) 512-0909. 

Scope of use 

Reading Mastery is appropriate for elementary-age children who 
are above, at, or below grade level in their reading performance. 
The program also can be used with English language learners 
and special education students. 



Cost 

Student materials include storybooks (grades pre-K-1) or text- 
books (grades 2-6), workbooks, and test books. The cost per 
student ranges from $200 to $300 for the first year of implemen- 
tation. A full set of teaching materials— a one-time purchase- 
costs between $650 and $1,000 for each grade level. Additional 
components include literature collections. Independent Readers, 
seatwork blackline masters, and Practice and Review CD-ROMs 
for student practice of skills taught in the program. SRA Teach- 
ing Tutor CD-ROMs supplement consultant-led professional 
development. Additional information on costs of training materials 
and workshops is available online. 



4. These numbers show the average and range of student-level improvement indices for all findings across the study. 

5. This program is also known as Direct Instruction using the Reading Mastery texts or SRA Direct Instruction— Reading Mastery. 
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Additional program Teaching 

information a typical so- to 45-minute Reading Mastery lesson includes 
(continued) seven to nine short activities that encompass multiple strands 
of content, such as phonemic awareness, letter-sound cor- 
respondence, sounding out words, word recognition, vocabulary, 
oral reading fluency, or comprehension. The teaching routine 



repeated throughout the curriculum is composed of the following 
steps: modeling new content, providing guided practice, and 
implementing individual practice and application. Lesson scripts 
act as a guide for teachers. Signals and group responses are 
used to keep students involved and on task, and to control 
lesson pacing. The program typically spans one academic year. 



Research 



A total of 175 studies reviewed by the WWC investigated the 
effects of Reading Mastery on adolescent learners. One study 
(Stockard, 2010) is a randomized controlled trial that meets 
WWC evidence standards, and one study (Yu & Rachor, 2000) 
is a quasi-experimental design that meets WWC evidence stan- 
dards with reservations. The remaining 173 studies do not meet 
either WWC evidence standards or eligibility screens. 

Meets evidence standards 

Stockard (2010) conducted a randomized controlled study that 
examined the effects of Reading Mastery Signature Edition on 
4th graders in an elementary school in the midwestern United 
States. General education students were randomly assigned to 
a treatment or control condition using alternative assignment 
with random start technique (see Appendix A1.1 for more details 
on the random assignment method). Two pairs of teachers were 
randomly assigned to a treatment or control condition via a coin 
flip. The WWC based its effectiveness ratings on findings from 
comparisons of the 29 students in two classrooms who received 
Reading Mastery Signature Edition and the 28 control students 
in two classrooms who received the Scott Foresman Basai 
Reading Program. The study reported students’ outcomes after 
five months of program implementation. 



Meets evidence standards with reservations 

Yu and Rachor (2000) conducted a retrospective quasi-experi- 
mental study that examined the effects of Reading Mastery on 
students from three grades in six schools in the northwestern 
United States (three schools implemented Reading Mastery and 
three did not). Each of the three Reading Mastery schools was 
matched (based on poverty level and percentage of minority 
students) with a comparison school that was not using Reading 
Mastery. Then, Reading Mastery students were matched to 
comparison-group students within each grade level on the basis 
of reading scores and demographic characteristics. The WWC 
based its effectiveness rating on findings from students in two 
grades.® The 4th-grade group consisted of 71 students who 
received Reading Mastery and 71 students in the comparison 
group who received standard reading instruction. The 5th-grade 
group consisted of 81 students in the Reading Mastery group 
and 81 students in the comparison group. The study reported 
students’ outcomes after one and two years of program 
implementation.^ 



6. Findings for 3rd-grade students are outside the scope of the Adolescent Literacy review. 

7. Two-year findings are considered for the effectiveness rating because these findings reflect the maximum exposure to the program. One-year findings 
are not included in this rating but are reported in Appendix A4. 
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Research (continued) Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or medium to large (see the WWC Procedures and Stan- 
dards Handbook, Appendix G). The extent of evidence takes into 
account the number of studies and the total sample size across 
the studies that meet WWC evidence standards with or without 
reservations.® 



The WWC considers the extent of evidence for Reading 
Mastery to be small for the reading fluency and comprehension 
domains for adolescent learners. No studies that meet WWC 
evidence standards with or without reservations examined the 
effectiveness of Reading Mastery on adolescent learners in the 
alphabetics or general literacy achievement domains. 



Effectiveness Findings 

The WWC review of Adolescent Literacy interventions addresses 
student outcomes in four domains: alphabetics, reading fluency, 
comprehension, and general literacy achievement. The studies 
included in this report cover two domains: reading fluency 
and comprehension. The findings below present the authors’ 
estimates and WWC-calculated estimates of the size and the 
statistical significance of the effects of Reading Mastery on 
adolescent learners.® 

Reading fiuency. Stockard (2010) found statistically signifi- 
cant positive effects of Reading Mastery on the AIMS Web 
Curriculum-Based Measurement Words Read Correct for 4th 
graders. According to WWC calculations, the effect was not 
statistically significant, but it was large enough to be considered 
substantively important (i.e., an effect size of at least 0.25). Thus, 
for the reading fluency domain, one study showed substantively 
important positive effects. 

Comprehension. Yu and Rachor (2000) found a statistically 
significant positive effect of Reading Mastery on the Riverside 



Publishing Off Grade Reading Proficiency test for the grade 4 
cohort, and statistically significant negative effects on the State 
Reading Proficiency test for the grade 5 cohort. According to 
WWC calculations, the effects were not statistically significant, 
and the average effect across two cohorts was not large enough 
to be considered substantively important.''® Thus, for the com- 
prehension domain, one study showed indeterminate effects. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings, the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Procedures and Standards Handbook, 
Appendix E). 



8. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on 
the number and size of studies. Additional factors associated with a related concept-external validity, such as the students’ demographics and the 
types of settings in which studies took place-are not taken into account for the categorization. Information about how the extent of evidence rating was 
determined for Reading Mastery is in Appendix A6. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and 
Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of 
Stockard (2010), no corrections for multiple comparisons were needed. For the Reading Mastery studies summarized here, corrections for clustering 
were needed, so the significance levels may differ from those reported in the original studies. 

10. The WWC computes an average effect size as a simple average of the effect sizes across all individual findings within the study domain. 
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The WWC found Reading 
Mastery to have potentially 
positive effects on reading 
fluency and no discernible 
effects on comprehension 
for adolescent learners 



Improvement index 

The WWC computes an improvement index for each 
individual finding. In addition, within each outcome domain, 
the WWC computes an average improvement index for each 
study and an average improvement index across studies (see 
WWC Procedures and Standards Handbook, Appendix F). The 
improvement index represents the difference between the per- 
centile rank of the average student In the Intervention condition 
and the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is entirely based on the size of the effect, regardless of 
the statistical significance of the effect, the study design, or the 
analysis. The improvement index can take on values between 
-50 and +50, with positive numbers denoting favorable results 
for the intervention group. 



The Improvement Index for reading fluency is +19 percentile 
points for a single finding from one study. The average improve- 
ment index for comprehension is -7 percentile points, with a range 
of -20 to +7 percentile points across findings from one study. 

Summary 

The WWC reviewed 175 studies on Reading Mastery for ado- 
lescent learners. One of these studies meets WWC evidence 
standards, and one study meets WWC evidence standards 
with reservations; the remaining 173 studies do not meet either 
WWC evidence standards or eligibility screens. Based on these 
studies, the WWC found potentially positive effects on reading 
fluency and no discernible effects on comprehension for 
adolescent learners. The conclusions presented in this report 
may change as new research emerges. 



R6f6r6nC6S Meets WWC evidence standards 

Stockard, J. (2010). Fourth graders’ growth in reading fluency: A 
pretest-posttest randomized control study comparing Read- 
ing Mastery and Scott Foresman Basal Reading Program. 
Eugene, OR: National Institute for Direct Instruction. 

Meets WWC evidence standards with reservations 

Yu, L., & Rachor, R. (2000, April). The two-year evaluation of 
the three-year Direct Instruction program, in an urban public 
school system. Presented at the annual meeting of the Ameri- 
can Educational Research Association, New Orleans, LA. 

Studies that fall outside the Adolescent Literacy review 

protocol or do not meet WWC evidence standards 

Adams, G., &Carnlne, D. (2003). Direct Instruction. In H. L. 
Swanson, K. R. Harris, &S. Graham (Eds.), Handbook of 
learning disabilities (pp. 403-416). New York: Guilford Press. 
The study is ineligible for review because it is not a primary 
analysis of the effectiveness of an intervention, such as a 
meta-analysis or research literature review. 



Adams, G. L., & Englemann, S. (1996). Research on Direct 
Instruction: 25 years beyond DISTAR. Seattle, WA: Educa- 
tional Achievement Systems. The study is ineligible for 
review because it is not a primary analysis of the effective- 
ness of an intervention, such as a meta-analysis or research 
literature review. 

Altwerger, B., Arya, P., Jin, L., Jordan, N. L., Laster, B., Martens, 
R, ... Wiltz, N. (2004). When research and mandates collide: 
The challenges and dilemmas of teacher education in the 
era of NCLB. English Education, 36(2), 119-133. The study is 
ineligible for review because it does not use a sample aligned 
with the protocol— the sample is not within the specified age 
or grade range. 

Altwerger, B., Arya, R, Jin, L., Lang, D., Laster, B., Martens, R, 

... Wilson, G. P. (2001, December). The impact of reading 
programs on second graders’ reading. Symposium conducted 
at the annual meeting of the National Reading Conference, 
Miami, FL. The study is ineligible for review because it does 
not use a sample aligned with the protocol— the sample is not 
within the specified age or grade range. 
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RsfsrsnCGS (continued) American Federation of Teachers. (1999). Building on the best, 

learning from what works: Five promising remedial reading 
intervention programs. Washington, DC: Author. The study is 
ineligible for review because it is not a primary analysis of the 
effectiveness of an intervention, such as a meta-analysis or 
research literature review. 

Additional source: 

American Federation of Teachers. (1998). Building on the best, 
learning from what works: Seven promising reading and 
English language arts programs. Washington, 

DC: Author. 

American Institutes for Research. (1999). An educators’ guide 
to schoolwide reform. Washington, DC: Author. The study is 
ineligible for review because it is not a primary analysis of the 
effectiveness of an intervention, such as a meta-analysis or 
research literature review. 

Ashworth, D. R. (1999). Effects of Direct Instruction and basal 
reading instruction programs on the reading achievement of 
second graders. Reading Improvement, 36(4), 150-156. The 
study is ineligible for review because it does not use a sample 
aligned with the protocol— the sample is not within the speci- 
fied age or grade range. 

Banta, K. (2002). Direct Instruction Reading Mastery: A small 
scale study focused on male students with specific learning 
disabilities. Does it really work? Unpublished master’s thesis, 
Cardinal Stritch University, Milwaukee, Wl. The study is 
ineligible for review because it does not use a sample aligned 
with the protocol— the sample includes less than 50% general 
education students. 

Bateman, B. (1991). Teaching word recognition to slow-learning 
children. Reading, Writing, and Learning Disabilities, 7, 1-16. 
The study is ineligible for review because it is not a primary 
analysis of the effectiveness of an intervention, such as a 
meta-analysis or research literature review. 

Borman, G. D., Flewes, G. M., Overman, L. T, & Brown, S. (2002). 
Comprehensive school reform and student achievement: 

A meta-analysis. Baltimore, MD: CRESPAR/Johns Flopkins 



University. The study is ineligible for review because it is not a 
primary analysis of the effectiveness of an intervention, such 
as a meta-analysis or research literature review. 

Brandes, J. A. (2004). Literacy instruction for students with 
autism spectrum disorder (ASD): A national survey of special 
educators (Doctoral dissertation, University of Oklahoma, 
2004). Dissertation Abstracts International, 65(07A), 230-2558. 
The study is ineligible for review because it does not use a 
sample aligned with the protocol— the sample includes less 
than 50% general education students. 

Brattland, E. M. (1991). Oral strategies to develop vocabulary in 
Reading Mastery III. Unpublished master’s project, Weber 
State University, Ogden, UT. The study is ineligible for review 
because it does not use a comparison group design or a 
single-case design. 

Brent, G., & Diobilda, N. (1993). Effects of curriculum alignment 
versus direct instruction on urban children. Journal of Educa- 
tional Research, 86, 333-338. The study is ineligible for review 
because it does not use a sample aligned with the protocol— 
the sample is not within the specified age or grade range. 

Briggs, K. L., & Clark, C. (1997). Reading programs for students 
in the lower elementary grades: What does the research say? 
Austin, TX: Texas Center for Educational Research. The study 
is ineligible for review because it is not a primary analysis of 
the effectiveness of an intervention, such as a meta-analysis 
or research literature review. 

Brooks, G. (1999). What works for slow readers? Support for 
Learning, 74(1), 27. The study is ineligible for review because 
it is not a primary analysis of the effectiveness of an interven- 
tion, such as a meta-analysis or research literature review. 

Butler, M. T. (2001). Comparison of the effects of direct instruc- 
tion and basal instruction on the reading achievement of first- 
grade students identified as students with reading difficulties 
(Doctoral dissertation, University of Alabama, 2001). Disserta- 
tion Abstracts International, 62(09A), 203-3002. The study Is 
ineligible for review because it does not use a sample aligned 
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Appendix 



Appendix A1.1 Study characteristics: Stockard, 2010 



Characteristic 


Description 


Study citation 


Stockard, J. (201 0). Fourth graders’ growth in reading fiuency: A pretest-posttest randomized controi study comparing Reading Mastery and Scott Foresman Basal Reading 
Program. Eugene, OR: National Institute for Direct Instruction. 


Participants 


In this randomized study, 58 general education elementary students were assigned to a treatment or control condition, using alternative assignment with a random start 
technique. The author used a class list (arranged in alphabetical order) to conduct the assignment. First, the author used a random numbers table to determine where to begin 
in the class list. Second, the author used a coin flip to determine whether the assignment would start with the treatment or control group. Finally, after this initial assignment 
was determined, the author assigned each additional student in an alternating fashion to the treatment or control group. For example, in the first step, the author might have 
started the assignment with John Smith (based on a random numbers table). In the second step, the author might have assigned John Smith to the treatment group (based 
on the coin flip). Then, the author would have proceeded through the rest of the ordered class list (alternating between the control and treatment groups). Four classrooms 
participated in the study. Two pairs of teachers were formed and then — within these pairs — teachers were randomly assigned to the treatment or control group via a coin flip.^ 
Students were predominantly non-Hispanic whites from middle-income families. The analysis sample consisted of 29 fourth-grade students who received Reading Mastery 
and 28 fourth-grade students in the comparison group. 


Setting 


The study was conducted in a midwestern elementary school. 


Intervention 


Beginning in the fall of 2009, students in the treatment condition received instruction for 90 minutes a day in the SRA/McGraw-Flill program, Reading Mastery Signature Edition. 
Students were exposed to Reading Mastery om five months. 


Comparison 


The control group received instruction for 90 minutes a day in the Scott Foresman Basal Reading Program, which the school had been using in prior years. 


Primary outcomes 
and measurement 


Data on the AIMS Web Curriculum-Based Measurement Words Read Correct were gathered in the spring of 2009 before instruction began (for use as a baseline measure), 
in the fall of 2009 shortly after the start of the school year, and again in winter of 201 0, approximately halfway through the school year. For a more detailed description of this 
outcome measure, see Appendix A2.1 . 


Staff or teacher training 


No information about training was provided. 



1. The study author did not describe how students assigned to the treatment group were assigned to the two treatment teachers or how students assigned to the controi group were assigned to 
the two controi teachers. 
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Appendix A1.2 Study characteristics: Yu & Rachor, 2000 



Characteristic 


Description 


Study citation 


Yu, L., & Rachor, R. (2000, April). The two-year evaluation of the three-year Direct Instruction program, in an urban public school system. Presented at the annual meeting ot 
the American Educational Research Association, New Orleans, LA. 


Participants 


This retrospective quasi-experiment included students in three elementary schools that participated in Reading Masterytor two consecutive years. Each ot the three Reading 
Mastery schools was matched with a school with a similar level ot poverty and percentage of minority students. Then, Reading Mastery students were matched with comparison 
students in the same grade on the basis of race, gender, free lunch status, and reading achievement test scores. This review focuses on findings from students who were in 
grades 4 and 5^ during the first year of program implementation (1997/98). The analysis sample for students who were in grade 4 in 1997/98 consisted of 71 students who 
received Reading Mastery and It matched comparison students. The analysis sample for students who were in grade 5 in 1997/98 consisted of 81 students in the Reading 
Mastery group and 81 matched comparison students. Of the students, more than 96% were African-American, and about 80% were eligible for free or reduced-price lunch. 
The study reported students' outcomes after two years of program implementation; these findings were used in the intervention ratings and can be found in Appendix A3. 2. 
Additional findings reflecting students' outcomes after one year of program implementation can be found in Appendix A4. 


Setting 


The study was conducted in six elementary schools in a northwestern urban public school system. 


Intervention 


The complete Reading Mastery program was adopted, including all materials and teacher training. The program provided scripted, carefully sequenced lessons; rapid pacing; 
and responses of the students in unison as well as in individual turns. Students were exposed to Reading Mastery over two school years. 


Comparison 


The comparison group received the standard instruction provided in the regular school curriculum. 


Primary outcomes 
and measurement 


For both the pretest and posttest, students took the State Reading Proficiency Test and the Riverside Publishing Off Grade Reading Proficiency Test. The State Reading 
Proficiency Test provided data at pretest and after two years of intervention implementation for the 5th-grade group and data after one year of intervention implementation for 
the 4th-grade group. The Riverside Publishing Off Grade Reading Proficiency Test provided data at pretest and after two years of intervention implementation for the 4th-grade 
group, and data after one year of intervention implementation for the 5th-grade group. For a more detailed description of these outcome measures, see Appendix A2.2. 


Staff or teacher training 


Teachers participated in training prior to implementation and were provided with ongoing consultations from the provider for the program duration. No additional details about 
training were provided. 



1. Findings for 3rd grade are outside the scope of the Adoiescent Literacy review. 
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Appendix A2.1 Outcome measures for the reading fluency domain 



Outcome measure 


Description 


AIMS Web Curriculum- 
Based Measurement 
Words Read Correct 


When a child reads graded passages aloud for one minute, this curriculum-based AIMS web assessment gathers data on reading fluency in the form of number of words read 
correctly. The assessment has well-established validity and reliability. AIMS Web is a benchmark and progress-monitoring system based on direct, frequent, and continuous 
student assessment (as cited in Stockard, 2010). 



Appendix A2.2 Outcome measures for the comprehension domain 



Outcome measure 


Description 


Reading comprehension construct 


Riverside Publishing 
Off Grade Reading 
Proficiency Test 


This test for 3rd- and 5th-grade students is designed to measure students' reading proficiency. Test items, which are based on fiction, nonfiction, and poetry reading pas- 
sages, are designed to capture four strands of learning outcomes defined by the publisher as (1) examining meaning given a fiction or poetry text (examining meaning indicates 
that students are able to comprehend the overall meaning of what they have read), (2) extending meaning given a fiction or poetry text (extending meaning indicates that 
students can interpret what they have read and infer beyond the text), (3) examining meaning given a nonfiction text, and (4) extending meaning given a nonfiction text (as 
cited in Yu & Rachor, 2000). 


State Reading 
Proficiency Test 


This state annual reading test is administered to 4th- and 6th-grade students (as cited in Yu & Rachor, 2000). No additional information on the test was provided. 
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Appendix A3.1 Summary of study findings inciuded in the rating for the reading fluency domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
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Reading 

Mastery Comparison 
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Mean 

difference^ 
{Reading Mastery 
- comparison) 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 







Stockard, 2010^’» 










AIMS Web Grade 4 4/57 

CBM Words 
Read Correct 


149.0 

(29,5) 


134.7 

(27,8) 


14,30 


0,49 


ns 


-Hi 9 


Oomain average for reading fluency (Stockard, 2010)^ 








0.49 


ns 


+19 



ns = not statistically significant 

AIMS Web CBM = AIMS Web Curriculum-Based Measurement 



1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the reading fluency domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect-size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -r50, with positive numbers denoting favorable results for the intervention group. 

7. The Reading Mastery group mean outcome values for Stockard (2010) are the unadjusted control group posttest means plus the difference in mean gains between the intervention and control 
groups. Control group means are unadjusted. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Stockard (2010), a correction for clustering was needed, so the significance levels may differ from those reported in the original 
study. 

9. This row provides the study average, which, in this instance, is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. 

The domain improvement index is calculated from the average effect size. 
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Appendix A3.2 Summary of study findings inciuded in the rating for the comprehension domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


Reading 

Mastery Comparison 

group group 


Mean 

difference^ 
{Reading Mastery 
-comparison) 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 



Yu & Rachor, 2000^« 



Riverside Publishing Off Grade 
Reading Proficiency Test 


Grade 4 6/142 


209.01 

(20,16) 


205.46 

(18.42) 


3.55 


0.18 


ns 


+7 


State Reading Proficiency Test 


Grades 6/162 


201.15 

(25.30) 


213.80 

(23,42) 


-12,65 


-0.52 


ns 


-20 


Domain average for comprehension (Yu & Rachor, 2000)^ 








-0.17 


na 


-7 



ns = not statistically significant 
na = not applicable 



1. This appendix reports two-year findings considered for the effectiveness rating and the average improvement indices for the comprehension domain. One-year findings from Yu and Rachor 
(2000) are not included in these ratings but are reported in Appendix A4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect-size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

7. The Reading Mastery group mean outcome values for Yu and Rachor (2000) differ from those presented in the paper. The WWC calculated the program group mean using a difference-in- 
differences approach (see WWC Handbook)— calculating the program means by adding the impact of the program (i.e., difference in mean gains between the intervention and control groups) to 
the unadjusted control group posttest means. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Yu and Rachor (2000), a correction for clustering was needed, so the significance levels may differ from those reported in the 
original study. 

9. This row provides the study average, which, in this instance, is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. 

The domain improvement index is calculated from the average effect size. 
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Appendix A4 Summary of one-year implementation findings for the comprehension domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation)^ 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


Reading 

Mastery Comparison 

group group 


Mean 

difference^ 
{Reading Mastery 
- comparison) 


Statistical 

Effect significance^ 

size^ (at a = 0.05) 


Improvement 

index^ 



Yu & Rachor, 2000^’» 



State Reading Proficiency Test 


Grade 4 


6/142 


207.13 

(12.28) 


200.92 

(12.18) 


6,21 


0.51 


ns 


-Hi 9 


Riverside Publishing Off Grade 
Reading Proficiency Test 


Grade 5 


6/162 


207.42 

(21,33) 


204,95 

(24,86) 


2.47 


0,11 


ns 


-h4 



ns = not statistically significant 



1. This appendix presents one-year findings for measures that fall in the comprehension domain. Two-year findings were used for rating purposes and are presented in Appendix A3. 2. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect-size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and -r50, with positive numbers denoting results favorable to the intervention group. 

7. The Reading Mastery group and control group mean outcome values for Yu and Rachor (2000) are the unadjusted one-year posttest means. 

8. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards 
Handbook, Appendix D for multiple comparisons. In the case of Yu and Rachor (2000), a correction for clustering was needed, so the significance levels may differ from those reported in the 
original study. 
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Appendix A5.1 Reading Mastery rating for the reading fluency domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the reading fiuency outcome domain, the WWC rated Reading Mastery as having potentially positive effects for adolescent learners. The remaining ratings (mixed 
effects, no discernible effects, potentially negative effects, or negative effects) were not considered, as Reading Mastery was assigned the highest applicable rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. One study showed substantively important positive effects. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing 
indeterminate effects than showing statistically significant or substantively important pos/f/Ve effects. 

Met. No studies showed statistically significant or substantively important negative effects. No studies showed indeterminate effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies showed statistically significant positive effects. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies showed statistically significant or substantively important negative effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5.2 Reading Mastery rating for the comprehension domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentiaiiy negative, or negativeJ 
For the comprehension outcome domain, the WWC rated Reading Mastery as having no discernible effects for adolescent learners. 



Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: No studies showing a statistically significant or substantively important effect, either positive or negative. 

Met. No studies showed a statistically significant or substantively important effect, either positive or negative. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies showed a statistically significant positive effect. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies showed a statistically significant or substantively important negative effect. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. No studies showed a statistically significant or substantively important positive effect. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing 
indeterminate effects than showing statistically significant or substantively important pos/f/Ve effects. 

Not met. No studies showed a statistically significant or substantively important negative effect. One study showed indeterminate effects. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important pos/f/Ve effect. 

Not met. No studies showed a statistically significant or substantively important positive effect, and no studies showed a statistically significant or 
substantively important negative effect. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. No studies showed a statistically significant or substantively important effect. One study showed indeterminate effects. 



1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 



(continued) 
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Appendix A5.2 Reading Mastery rating for the comprehension domain (continued) 



Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: One study showing a statisticaily significant or substantively important negative effect and no studies showing a statisticaiiy significant or substantively 
important positive effect. 

Not met. No studies showed a statistically significant or substantiveiy important effect, either positive or negative. 

OR 

• Criterion 2: Two or more studies showing statistically significant or substantiveiy important negative effects, at least one study showing a statistically significant 
or substantiveiy important positive effect, and more studies showing statistically significant or substantiveiy important negative effects than showing statistically 
significant or substantively important positive effects. 

Not met. No studies showed a statistically significant or substantively important effect, either positive or negative. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies showed a statistically significant negative effect. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. No studies showed a statistically significant or substantively important positive effect. 
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Appendix A6 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schools 


Sample size 


Students 


Extent of evidence^ 


Alphabetics 


na 


na 




na 


na 


Reading fluency 


1 


1 




57 


Small 


Comprehension 


1 


6 




304 


Small 


General literacy achievement 


na 


na 




na 


na 



na = not applicable/not studied 

1. A rating of “medium to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. Other- 
wise, the rating is “small.” For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 
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